矩阵求导

Matrix Derivative

矩阵的导数运算

分子： $f (y)$ 为 $1 \times 1$ 的标量；分母： $y$ 为 $n \times 1$ 的向量

与分母的行数相同

\begin{array}{r} \frac{\partial f (y)}{\partial y} = (\begin{array}{c} \frac{\partial f (y)}{\partial y_{1}} \\ \frac{\partial f (y)}{\partial y_{2}} \\ ⋮ \\ \frac{\partial f (y)}{\partial y_{n}} \end{array}) \end{array}

与分子的行数相同

\begin{array}{r} \frac{\partial f (y)}{\partial y} = (\begin{array}{c} \frac{\partial f (y)}{\partial y_{1}}, \frac{\partial f (y)}{\partial y_{2}}, \dots, \frac{\partial f (y)}{\partial y_{n}} \end{array}) \end{array}

注意

以上两种布局没有本质区别，互为转置；为方便起见，以下的推导默认使用分母布局。

\begin{array}{r} f (u) = (\begin{array}{c} f_{1} (u) \\ f_{2} (u) \\ \dots \\ f_{m} (u) \end{array}) \Rightarrow \frac{\partial f (u)}{\partial u} = (\begin{array}{c} \frac{\partial f_{1} (u)}{\partial u} \frac{\partial f_{2} (u)}{\partial u} & \dots & \frac{\partial f_{m} (u)}{\partial u} \end{array}) \end{array}

f (y) = {(\begin{matrix} f_{1} (y) \\ f_{2} (y) \\ ⋮ \\ f_{m} (y) \end{matrix})}_{(m \times 1)} y = {(\begin{matrix} y_{1} \\ y_{2} \\ ⋮ \\ y_{n} \end{matrix})}_{(n \times 1)}

分母布局的结果如下：（其转置（分子布局）实际上为雅可比矩阵）

\begin{array}{r} \frac{\partial f (y)}{\partial y} = {(\begin{array}{c} \frac{\partial f (y)}{\partial y_{1}} \\ \frac{\partial f (y)}{\partial y_{2}} \\ ⋮ \\ \frac{\partial f (y)}{\partial y_{n}} \end{array})}_{(n \times 1)} = {(\begin{array}{c} \frac{\partial f_{1} (y)}{\partial y_{1}} & \frac{\partial f_{2} (y)}{\partial y_{1}} & \dots & \frac{\partial f_{m} (y)}{\partial y_{1}} \\ \frac{\partial f_{1} (y)}{\partial y_{2}} & \frac{\partial f_{2} (y)}{\partial y_{2}} & \dots & \frac{\partial f_{m} (y)}{\partial y_{2}} \\ \dots & \dots & \dots & \dots \\ \frac{\partial f_{1} (y)}{\partial y_{n}} & \frac{\partial f_{2} (y)}{\partial y_{n}} & \dots & \frac{\partial f_{m} (y)}{\partial y_{n}} \end{array})}_{(n \times m)} \end{array}

u = {(\begin{matrix} u_{1} \\ u_{2} \\ ⋮ \\ u_{n} \end{matrix})}_{(n \times 1)} y = {(\begin{matrix} y_{1} \\ y_{2} \\ ⋮ \\ y_{n} \end{matrix})}_{(n \times 1)}

\begin{array}{r} \frac{\partial (u^{T} f)}{\partial u} = f \end{array}

\begin{aligned} \frac{\partial A y}{\partial y} & = A^{T} \end{aligned}

二次型的求导：

\begin{array}{r} \frac{\partial y^{T} A y}{\partial y} = A y + A^{T} y \end{array}

如果 $A$ 为对称阵，则有 $\frac{\partial y^{T} A y}{\partial y} = 2 A y$

标量的链式求导，注意矩阵不满足乘法交换律
使用分母布局：

\begin{array}{r} \frac{\partial J}{\partial \vec{u}} = \frac{\partial y}{\partial \vec{u}} \frac{\partial J}{\partial y} \end{array}

离散状态空间方程： ${\vec{x}}_{[k + 1]} = A {\vec{x}}_{[k]} + B {\vec{u}}_{[k]}$

代价函数： $J = {\vec{x}}_{[k + 1]}^{T} {\vec{x}}_{[k + 1]}$

\frac{\partial J}{\partial \vec{u}} = \frac{\partial {\vec{x}}_{[k + 1]}}{\partial {\vec{u}}_{[k]}} \frac{\partial J}{\partial {\vec{x}}_{[k + 1]}} = B^{T} \cdot 2 {\vec{x}}_{[k + 1]}

\begin{aligned} \hat{z} & = y_{1} + y_{2} x J = \sum_{i = 1}^{n} {[z_{i} - (y_{1} + y_{2} x_{i})]}^{2} \end{aligned}

使用矩阵求解

\vec{z} = (\begin{matrix} z_{1} \\ z_{2} \\ ⋮ \\ z_{n} \end{matrix}) x = (\begin{matrix} 1 & x_{1} \\ 1 & x_{2} \\ ⋮ & ⋮ \\ 1 & x_{n} \end{matrix}) y = (\begin{matrix} y_{1} \\ y_{2} \end{matrix}) \vec{\hat{z}} = \vec{x} y = (\begin{matrix} y_{1} + y_{2} x_{1} \\ y_{1} + y_{2} x_{2} \\ ⋮ \\ y_{1} + y_{2} x_{n} \end{matrix})

\begin{aligned} \vec{z} - \vec{\hat{z}} & = (\begin{array}{c} z_{1} - (y_{1} + y_{2} x_{1}) \\ z_{2} - (y_{1} + y_{2} x_{2}) \\ ⋮ \\ z_{n} - (y_{1} + y_{2} x_{n}) \end{array}) \\ J & = [\vec{z} - \vec{\hat{z}}]^{T} [\vec{z} - \vec{\hat{z}}] \\ = [{\vec{z}}^{T} - y^{T} {\vec{x}}^{T}] [\vec{z} - \vec{x} y] \\ = {\vec{z}}^{T} \vec{z} - y^{T} {\vec{x}}^{T} \vec{z} - {\vec{z}}^{T} \vec{x} y + y^{T} {\vec{x}}^{T} \vec{x} y \\ = {\vec{z}}^{T} \vec{z} - 2 {\vec{z}}^{T} \vec{x} y + y^{T} {\vec{x}}^{T} \vec{x} y \end{aligned}

\begin{aligned} \frac{\partial J}{\partial y} = - 2 ({\vec{z}}^{T} \vec{x}) + 2 {\vec{x}}^{T} \vec{x} y = 0 \Rightarrow y = ({\vec{x}}^{T} \vec{x})^{- 1} {\vec{x}}^{T} \vec{z} \end{aligned}

如果没有解析解，使用机器学习的梯度下降

定义初始值 $y^{*}$
循环迭代 $y^{*} = y^{*} - α \nabla$

梯度： $\nabla = \frac{\partial J}{\partial y}$
学习率： $α = (\begin{matrix} α_{1} & 0 \\ 0 & α_{2} \end{matrix})$
注意如果没有归一化处理，学习率的选取要结合未知向量各值的数量级
归一化处理后：各个未知数用同一个学习率，使用标量 $α$ 即可